Papers with attention module
Revisiting Multimodal Transformers for Tabular Data with Text Fields (2024.findings-acl)
Copied to clipboard
| Challenge: | Tabular data with text fields can be used in financial risk assessment and diagnosis prediction. |
| Approach: | They propose a tabular/text dual-stream Transformer network with numerical embedding schemes and an overall attention module to estimate whether a prediction is uncertain. |
| Outcome: | The proposed model can estimate whether a prediction is uncertain or not based on two well-informed modality streams . |
IMO: Greedy Layer-Wise Sparse Representation Learning for Out-of-Distribution Text Classification with Pre-trained Models (2024.acl-long)
Copied to clipboard
| Challenge: | IMO is a machine learning model that learns invariant features from unseen domains. |
| Approach: | They propose IMO: Invariant features Masks for Out-of-Distribution text classification to achieve OOD generalization by learning invariant feature masks. |
| Outcome: | The proposed model outperforms baseline models in various evaluation metrics and settings. |
Joint Chinese Word Segmentation and Part-of-speech Tagging via Multi-channel Attention of Character N-grams (2020.coling-main)
Copied to clipboard
| Challenge: | Chinese word segmentation and part-of-speech tagging can be performed in a sequential order . existing studies have shown that jointly performing them can be effective . |
| Approach: | They propose a character-based neural model enhanced by multi-channel attention of n-grams. |
| Outcome: | The proposed model outperforms baseline models on five benchmark datasets. |
Improving Compositional Generalization in Semantic Parsing (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Generalization of models to out-of-distribution data has sparked substantial interest . compositional generalization is the ability to systematically generalize to test examples composed of components seen during training . |
| Approach: | They propose to extend compositional generalization in semantic parsing by using contextual representations and training attention to agree with pre-computed token alignments. |
| Outcome: | The proposed extensions improve compositional generalization on OOD compositions. |
ITA: Image-Text Alignments for Multi-Modal Named Entity Recognition (2022.naacl-main)
Copied to clipboard
| Challenge: | Recent work on Multi-modal Named Entity Recognition (MNER) relies on image information to model interactions between image and text representations. |
| Approach: | They propose to align image features into the textual space to better utilize attention mechanisms . they use regional object tags, captions and optical characters as visual contexts . |
| Outcome: | The proposed model can achieve state-of-the-art accuracy on multi-modal Named Entity Recognition datasets even without image information. |
When did you become so smart, oh wise one?! Sarcasm Explanation in Multi-modal Multi-party Dialogues (2022.acl-long)
Copied to clipboard
| Challenge: | Indirect speech achieves a constellation of discourse goals in human communication, but it is challenging for AI agents to comprehend such idiosyncrasies. |
| Approach: | They propose a task to generate natural language explanations of satirical conversations using a multimodal and code-mixed dataset to capture multimodality. |
| Outcome: | The proposed task generates natural language explanations of satirical conversations in a multimodal and code-mixed setting and surpasses baselines on almost all metrics. |
ATTEMPT: Parameter-Efficient Multi-task Tuning via Attentional Mixtures of Soft Prompts (2022.emnlp-main)
Copied to clipboard
| Challenge: | a new multi-task, parameter-efficient language model tuning method learns to transfer knowledge across different tasks via a mixture of soft prompts. |
| Approach: | They propose a multi-task, parameter-efficient language model tuning method that uses soft prompts to learn to transfer knowledge across different tasks. |
| Outcome: | The proposed method outperforms prompt tuning and outperfies or matches fully fine-tuned tuning approaches that use 10 times more parameters. |
Interpreting Positional Information in Perspective of Word Order (2023.acl-long)
Copied to clipboard
| Challenge: | Attention mechanism is a powerful and effective method utilized in natural language processing, but it is insensitive to positional information. |
| Approach: | They propose a weight concatenation operation to evaluate its efficacy in machine translation tasks. |
| Outcome: | The proposed operation can encode positional information and confirms our hypothesis. |
Chinese Couplet Generation with Syntactic Information (2022.coling-1)
Copied to clipboard
| Challenge: | Chinese couplet generation aims to generate a pair of clauses with certain rules adhered . previous studies have focused on learning the correspondences between antecedent and subsequent clauses . |
| Approach: | They propose to leverage syntactic information to generate Chinese couplets by POS tags and word dependencies. |
| Outcome: | The proposed approach outperforms baselines on a Chinese couplet generation dataset. |
Integrating Multi-scale Contextualized Information for Byte-based Neural Machine Translation (2024.findings-acl)
Copied to clipboard
| Challenge: | Subword tokenization is a common method for vocabulary building in NMT systems . but, it has its disadvantages, such as a vocabulary cannot be modified once it is learned . |
| Approach: | They propose a method that learns contextualized information of varying scales . they propose byte-based tokenization to solve these problems with few embedding parameters . |
| Outcome: | Experiments show that the proposed method outperforms subword-based methods in multilingual and out-of-domain scenarios. |
Progressively Guide to Attend: An Iterative Alignment Framework for Temporal Sentence Grounding (2021.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods to learn effective alignment between vision and language features are insufficient in practice due to complicated multi-step reasoning. |
| Approach: | They propose an iterative alignment network which iterates inter- and intra-modal features within multiple steps for more accurate grounding. |
| Outcome: | The proposed model performs better than the state-of-the-arts on three challenging benchmarks. |
Joint Chinese Word Segmentation and Part-of-speech Tagging via Two-way Attentions of Auto-analyzed Knowledge (2020.acl-main)
Copied to clipboard
| Challenge: | Chinese word segmentation and part-of-speech tagging are important fundamental tasks in natural language processing. |
| Approach: | They propose a neural model for Chinese word segmentation and part-of-speech tagging . they incorporate context features and syntactic knowledge for each input character . |
| Outcome: | The proposed model can learn and benefit from existing tools, but its quality may be poor. |
Context Compression for Auto-regressive Transformers with Sentinel Tokens (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing Transformer-based LLMs have limited performance due to complexity of attention module . key-value cache is the major memory footprint and inference latency problem . |
| Approach: | They propose a plug-and-play approach that incrementally compresses token activation into compact ones . they also profile the benefit of context compression on improving the system throughout . |
| Outcome: | The proposed approach reduces memory footprint and inference latency by compressing tokens into compact ones. |
PermuteFormer: Efficient Relative Position Encoding for Long Sequences (2021.emnlp-main)
Copied to clipboard
| Challenge: | Existing Transformers that scale to long sequences are not compatible with relative position encoding. |
| Approach: | They propose a Performer-based model with relative position encoding that scales linearly on long sequences. |
| Outcome: | The proposed model outperforms performer on long sequences with no computational overhead and outperformed vanilla Transformer on most of the tasks. |
InstructEd: Soft-Instruction Tuning for Model Editing with Hops (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing methods for model editing are limited due to excessive memorization and knowledge conflict issues. |
| Approach: | They propose to insert soft instructions into the attention module to facilitate interactions between instructions and questions and to understand and utilize new facts. |
| Outcome: | The proposed method achieves 10% improvement in one-hop (multi-hop) model editing on three datasets with LLaMAs and GPT2 . |
HATA: Trainable and Hardware-Efficient Hash-Aware Top-k Attention for Scalable Large Model Inference (2025.findings-acl)
Copied to clipboard
Ping Gong, Jiawei Yi, Shengnan Wang, Juncheng Zhang, Zewen Jin, Ouxiang Zhou, Ruibo Liu, Guanbin Xu, Youhui Bai, Bowen Ye, Kun Yuan, Tong Yang, Gong Zhang, Renhai Chen, Feng Wu, Cheng Li
| Challenge: | Existing top-k attention methods struggle to strike a balance between efficiency and accuracy. |
| Approach: | They propose a top-k attention approach that integrates low-overhead techniques into the Top-k Attention process to achieve 7.2 speedup compared to vanilla full attention. |
| Outcome: | The proposed approach achieves 7.2 speedup compared to current top-k attention methods while maintaining model accuracy. |